Distributed Stochastic Nested Optimization for Emerging Machine Learning Models: Algorithm and Theory

Traditional machine learning models can be formulated as the expected risk minimization (ERM) problem: minw∈Rd Eξ [l(w; ξ)], where w ∈ Rd denotes model parameter, ξ represents training samples, l(·) is loss function. Numerous optimization algorithms, such stochastic gradient descent (SGD), have been developed to solve ERM problem. However, a wide range of emerging are beyond this class problems, model-agnostic meta-learning (Finn, Abbeel, and Levine 2017). Of particular interest my research nested (SNO) problem, whose objective function has structure. Specifically, I focusing on two instances kind compositional (SCO) which cover meta-learning, area-under-the-precision recall-curve optimization, contrastive self-supervised learning, etc., bilevel (SBO) applied hyperparameter neural network architecture search, etc. With emergence large-scale distributed data, user data generated mobile devices or intelligent hardware, it imperative develop algorithms for SNO (Distributed SNO). A significant challenge optimizing problems lies in that (hyper-)gradient biased estimation full gradient. Thus, existing when them suffer from slow convergence rates. In talk, will discuss recent works about SCO (Gao Huang 2021; Gao, Li, 2022) SBO (Gao, Gu, Thai 2022; Gao under both centralized decentralized settings, including algorithmic details reducing bias gradient, theoretical rate, practical applications, then highlight challenges future research.